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I.  BACKGROUND 


\V>^ 

Many  expensive  systems,  such  as  missiles,  are  produced \  _> oV' 

\  >>''  o  ■ 1 

\  v^^'n 

to  complete  one  task  only  and  they  must  be  certified  to  the  v^£>' 
purchaser  to  be  highly  reliable  even  though  they  may  not  be 
tested  directly.  In  such  a  case,  the  producer  will  test  in-  \i 
dividual  components  or  subsystems  of  the  system  and  obtain 
a  point  estimate  from  the  resulting  binomial  data:  xi  suc¬ 
cesses  out  of  n^  trials  for  l<i<k,  where  k  is  the  number  of 
components.  Further,  most  contracts  require  an  interval  es¬ 
timate  in  the  form  of  a  lower  confidence  bound  on  system 
reliability.  The  interval  estimator  is  much  harder  to  pro¬ 
duce:  Even  though  the  literature  contains  a  number  of  methods 

for  computing  interval  estimates  for  the  reliability  of  series 
systems,  these  methods  often  do  not  produce  realistic  bounds 
for  highly  reliable  systems  where  the  data  consist  of  dis¬ 
parate  component  sample  sizes  and  a  paucity  of  failures. 

Although  there  are  a  number  of  properties  of  lower  con¬ 
fidence  bounds  that  have  been  described  historically  in  the 


mathematical  literature  as  desireable,  unbiased,  uniformly 
most  accurate,  optimum,  exact,  consistent  [7] ,  the  methods 
presently  available  are  approximations  so  these  properties 
are  difficult  to  verify  except  for  special,  unrealistic  cases. 
Properties  which  are  desireable  and  that  can  be  tested  by 
computer  simulation  are:  a  method  produces  a  6-exact( 1-a) % 
lower  confidence  bound  if  in  the  long  run  (l-a)£  of  the  bounds 
are  less  than  the  true  reliability  plus  6  [7]  (6*0  corresponds 
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to  an  exact  bound) .  The  method  should  produce  bounds  that 
are  as  close  to  the  true  reliability  as  possible:  the  variance 
of  the  confidence  bounds  about  the  true  reliability  should 
be  minimal.  The  method  should  be  flexible  so  that  it  can  be 
applied  to  the  variety  of  cases  that  arise  in  practice:  The 
constraints  of  time  and  expense  often  result  in  test  data  from 
a  variety  of  sample  sizes,  n^,  l<i<_k,  with  the  sample  sizes 
of  all  or  many  components  small  either  absolutely  or  with  re¬ 
spect  to  the  other  sample  sizes.  The  method  should  be  appli¬ 
cable  when  some  or  all  of  the  components  experience  no  failures 
in  testing. 

In  [7j ,  work  was  focused  on  series  systems  as  this  is 
the  simplest  case  with  the  most  existing  lower  confidence 
bound  methods.  There  are  presently  many  methods  used  to 
compute  lower  confidence  bounds  for  series  systems: 
modified  maximum  likelihood  (MMLI)  introduced  by  Easterling 
[2j ;  approximately  optimal  (AO)  presented  by  Mann  and  Grubb 
£9]  ;  modified  log  gamma  (MLG)  introduced  by  Borsting  and 
Woods  [l]  and  recently  modified  by  Tomsky  and  by  the  authors 
to  remove  an  inconsistency  (see  [73  for  further  discussion 
of  these  methods);  the  likelihood  ratio  methods  discussed 
below.  However,  each  method  has  difficulties  in  some  prac¬ 
tical  cases,  so  the  concept,  of  6-e.xact  bounds  was  introduced 
'  M  to  allow  a  larger  class  of  methods  to  be  considered.  It 
was  shown  that  some  new  methods  which  are  6*. 01  90?  lower 
confidence  bounds  seem  to  be  preferrable  in  many  practical 
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cases . 

In  1965,  Madansky  [8}  presented  a  method  for  series  and 
parallel  systems  that  was  based  on  the  result,  due  to  Wilks, 
that  -21n(PQ)  is  distributed  asymptotically  as  x2  with  one 
degree  of  freedom.  In  1968,  the  method  was  extended  to  all 
coherent  systems  by  Myhre  and  Saunders  [llj  .  As  originally 
introduced,  this  method  had  difficulties  when  the  system  was 
so  reliable  or  the  samples  taken  so  small  that  some  components 
were  tested  without  a  failure  resulting.  As  this  is  often 
the  case  for  expensive  highly  reliable  systems,  such  as  mis¬ 
siles,  an  initial  attempt  to  correct  this  difficulty  was  made 
by  adjusting  the  number  of  failures  to  4/kn^  for  the  i-th 
component  when  there  were  no  actual  failures  out  of  n.^  trials 
for  the  i-th  component.  The  adjusted  binomial  data  is  then  used 
by  the  likelihood  ratio  method  resulting  in  a  method  referred 
to  as  LRL. 

In  the  past,  the  likelihood  ratio  method  has  used  the 
maximum  likelihood  estimate  of  the  reliability  of  the 
system  components.  However,  in  terms  of  total  squared  error 
loss,  Stein  found  [6]  that  one  could  improve  on  the  MLE  when 
estimating  three  or  more  parameters  from  independent  normal 
observations  and  Efron  and  Morris  have  shown  how  to  extend 
this  estimator  to  binomial  data  [6’J  .  By  using  Stein's  estimates 
of  the  component  reliability  in  the  likelihood  ratio  method, 
a  potentially  acceptable  method  was  produced  though  it  tended 
to  be  too  optimistic  in  certain  situations.  Since  Stein's 
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estimator  shrinks  the  estimated  value  parameters  from  the  MLE 
toward  the  mean  of  the  prior  distribution,  it  gives  poor  es¬ 
timates  of  the  parameter  of  individual  components  that  have 
unusually  large  or  small  reliability.  What  is  needed  for  such 
cases  is  a  compromise  between  the  Stein's  estimator  and  the  MLE. 
That  compromise  consists  of  rules  which  limit  the  amount  of 
deviation  the  Stein's  estimator  can  take  from  the  MLE;  these 
rules  are  known  as  limited  translation  rules  [4].  These  rules 
reduce  the  total  risk  that  results  from  using  the  MLE,  but  have 
a  greater  total  risk  than  Stein's  estimator.  Therefore,  the 
savings  of  risk  of  a  limited  translation  rule  lies  between 
0,  for  the  MLE,  and  1,  for  the  Stein's  estimator.  In  our 
previous  report,  LRS13  was  introduced,  it  being  a  limited 
translation  version  of  Stein's  estimator  with  Q0%  savings 
followed  by  the  likelihood  ratio  method. 

Since  the  component  test  data  is  binomially  distributed 
and  not  normally  distributed,  an  empirical  Bayesian  approach 
was  applied  directly  to  the  binomial  data,  assuming  a  3eta 
prior  distribution.  (See  [7]  for  more  details.)  The  adjusted 
empirical  Bayesian  binomial  test  data  is  used  in  the  likeli¬ 
hood  ratio  method  resulting  in  the  method  LRB9- 

In  our  previous  report,  it  was  shown  that  the  likelihood 
ratio  methods  LRS13  and  LRB9  have  great  promise  especially 
in  the  unequal  sample  size  case  but  also  for  the  small  equal 
sample  size  case,  exactly  the  most  troublesome  case  for  other 
methods.  LRS13  was  not  always  a  6-exact  bound,  but  since 
the  80 %  level  of  savings  used  in  LRS13  was  chosen  arbitrarily, 
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further  work  was  warrented  to  determine  if  there  was  a  level 
of  savings  that  would  produce  a  better  method  for  either  the 
equal  or  unequal  sample  case. 

While  the  previous  work  was  confined  to  the  series  systems, 
another  type  of  system,  referred  to  as  a  pseudo-series,  is  very 
important  as  it  can  be  used  to  estimate  the  reliability  of  a 
missile  shot  from  a  submarine  and  having  re-entry  bodies 
successfully  ejected  from  it.  Making  a  few  assumptions  included 
below,  this  might  be  modeled  as:  R=RERpSRssRTSRI1R.j.n-1//2Ry , 

where  R  is  system  reliability;  RE  is  the  probability  that  a 
missile  is  successfully  ejected  from  the  submarine;  Rps>Rsc> 

RrpS  are  the  probabilities  that  a  missile  which  enters  the  first, 
second,  or  third  boost  phase  will  not  fail  during  it  (natur¬ 
ally,  RpS  is  conditioned  on  the  completion  of  the  first  boost 
phase,  ...);  R-^  is  the  probability  that  a  missile  which  enters 
the  horizontal  deployment  phase  will  not  fail  before  the  pre¬ 
determined  time  of  the  first  re-entry  body  release;  R^  is 
the  probability  that  the  missile  continues  to  function  between 
the  time  of  the  (j-l)-th  and  j-th  planned  release  of  a  re¬ 
entry  body  (2<J<N,  where  N  is  the  number  of  re-entry  bodies 
on  the  missile) .  It  is  assumed  that  the  R^^  are  equal  for  all 
j  above,  so  R^j  is  replaced  by  Rj.  Ry  is  the  probability  of 
success  of  the  portion  of  the  missile  which  is  concerned  only 
with  the  ejection  of  a  single  re-entry  body,  and  Ry  is  assumed 
to  be  constant  for  all  re-entry  bodies.  Thus,  the  reliability 
function  is:  H(p)=H  p,ai,  where  a,=l  except  for  one  component. 

A  _  1  1  1 


If  the  set  of  missiles  has  been  purchased  from  different 
vendors  or  gone  through  re-designing,  then  the  reliability  func- 

TT1 

tion  will  be  H(p)=S  co.II  p  °iJ.  Here,  w.  represents  the 

j=l  Ji-1  i  J 

proportion  of  missiles  from  vendor  j,  for  instance.  So  far, 
only  MLG  and  the  likelihood  ratio  methods  have  been  shown 
to  work  for  pseudo-series. 

In  the  past  few  months ,  we  have  concentrated  on  two 
problems:  1)  What  is  the  best  level  of  savings  to  use  in 

a  likelihood  ratio  method  such  as  LRS13?  Is  Q0%  optimum  in 
some  cases,  or  can  the  method  be  improved  by  using  50%,  65%, 
or  95%  savings?  2)  Among  all  methods  of  estimating  lower 
confidence  bounds  for  the  reliability  of  pseudo-series,  which 
are  best?  Do  the  likelihood  ratio  methods  that  worked  well 
for  series  work  as  well  for  pseudo-series? 

II.  METHODS  OF  SIMULATION  AND  COMPARISON 

Our  simulations  were  done  as  explained  in  our  prelimi¬ 
nary  report.  As  was  explained  there,  we  designate  a  number, 
k,  of  components  which  are  in  a  structure  (in  our  case,  a 
series  or  pseudo-series  structure) .  For  each  component  i, 
l£i<k,  we  assign  a  reliability  p^  and  specify  the  number  n.^ 
of  parts  to  be  tested.  We  then  simulate  success-failure  data 
for  each  component  and  compute  lower  confidence  bounds  on  the 
system  reliability  using  various  confidence  bound  techniques. 
This  process  is  repeated  100  times  and  the  results  are  recorded 
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for  each  lower  confidence  bound  method  which  was  considered. 

For  the  cases  with  series  systems,  we  compared  six  dif¬ 
ferent  limited  translation  versions  of  Stein's  estimator  which 
adjusted  data  and  fed  it  into  the  likelihood  ratio  method.  In 
our  previous  studies  of  series  systems,  this  seemed  a  promising 
method  of  estimating  lower  confidence  bounds.  For  the  pseudo¬ 
series  system,  we  considered  five  methods  which  had  seemed 
promising  (one  of  which  was  a  Stein's  method). 

After  the  lower  bounds  have  been  computed,  the  results  of 
each  lower  confidence  bound  method  are  compared  with  the  true 
system  reliability  (found  by  plugging  the  p^'s  into  the  reliabil¬ 
ity  function)  to  see  how  often  each  method  produced  a  confidence 
bound  which  was  no  greater  than  the  true  system  reliability. 

In  computing  90%  lower  confidence  bounds,  we  desire  no  more  than 
10%  of  the  computed  bounds  to  be  greater  than  the  true  system 
reliability.  We  also  consider  6-exact  bounds  by  comparing  the 
results  against  the  true  reliability  plus  5,  where  6=.01  and  .02. 

We  consider  how  close  the  results  of  a  method  are  by  com¬ 
puting  the  standard  deviation  of  the  100  results  about  the  true 
reliability.  We  also  compute  the  value  measured  about  the  true 
reliability  plus  6,  where  again  6=.01  and  .02.  In  our  original 
report,  we  considered  only  the  value  computed  about  the  true 
reliability,  but  we  feel  it  is  necessary  to  use  one  of  the  other 
values  when  a  method  seems  to  work  as  a  S=.01  or  .02  bound. 

In  our  preliminary  report,  we  compared  lower  confidence 
bound  methods  pairwise  by  comparing  the  type  of  bound  each  method 
produced  (an  exact  bound,  6=.01  or  .02  bound,  or  none  of  these). 
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and  the  standard  deviation  of  each  method's  results  about  the 
true  system  reliability.  We  also  counted  the  number  of  sets 
of  100  simulations  where  each  method  produced  each  type  of  bound. 
Both  types  of  results  are  necessary  in  evaluating  and  comparing 
methods,  but  we  found  that  one  type  of  result  could  contradict 
the  indication  of  the  other  type.  Below  we  see  an  excerpt 
from  the  preliminary  report  which  is  part  of  Table  VIII  of  that 
report  and  which  shows  the  results  of  series  simulations  for 
systems  with  k=20  components.  We  see  that  with  pairwise  com¬ 
parisons,  the  method  LRS13  looked  like  the  best  method  beating 
out  every  other  method  (the  rightmost  column  tallies  how  many 
methods  it  beat  out  versus  the  number  it  lost  to:  5-0) . 

Looking  at  the  other  part  of  the  table  however,  we  see  that 
LRS13  was  not  a  bound  3  out  of  7  times . 


best 

overall 

6-exact  bound 
6=0(exact)  S=.01 

5  =  .  02 

not  a  bound 

AO 

6 

~~  1 

7 

K 

5 

1 

1 

4 

1 

1 

1 

LRS13 

4 

3 

1 

3 

LRB9 

3 

7 

_i 

Pairwise  Comparison 

of  Methods 

for 

Comparison  of  Individual 

the 

Sets  of  Simulations 

Methods  with  all  Others 

AO 

MMLI 

MLGB 

LRL 

LRS13 

LRB9 

AO 

- 

7-0 

2-5 

3-4 

3-4 

0-7 

1-4 

MMLI 

0-7 

- 

1-6 

2-5 

3-4 

0-7 

0-5 

MLGB 

5-2 

6-1 

- 

7-0 

1-6 

2-5 

3-2 

LRL 

4-3 

5-2 

0-7 

- 

2-5 

0-7 

2-3 

LRS15 

4-3 

4-3 

6-1 

5-2 

- 

4-3 

5-0 

LRB9 

7-0 

7-0 

5-2 

7-0 

1 

3-4 

4-1 

In  the  past  few  months,  we  have  spent  time  looking  for 
a  method  of  evaluating  and  comparing  methods  which  would  not 
produce  contradictory  results  as  the  previous  method  could. 

First  we  considered  specifying  each  method  as  some  specific 
type  of  bound  (exact,  6= . 01 , 6= . 02 ,  or  none  of  these)  by  com¬ 
paring  how  often  (the  number  of  sets  of  simulations)  it  pro¬ 
duced  each  type  of  bound.  If  a  method  produced  an  exact  bound 
at  least  90$  of  the  time,  we  called  the  method  an  exact  method. 
If  the  method  was  not  an  exact  method,  but  produced  either 
an  exact  or  a  6=.01  bound  at  least  90$  of  the  time,  we  called 
it  a  5=.01  method.  We  similarly  defined  a  <5=.02  method. 

If  a  method  could  not  be  called  any  of  these,  it  was  always 
considered  a  5=.02  method  for  comparison  purposes.  We  compared 
results  pairwise  by  considering  whether  each  of  the  2  methods 
produced  a  bound  at  least  as  good  as  the  one  it  was  specified 
as.  If  so,  we  compared  the  standard  deviations  corresponding 
to  the  specified  bounds  and  the  method  with  the  lower  value 
would  win.  If  neither  produced  the  specified  bound,  the  amalles 
again  won.  And  if  only  one  produced  its  specified  bound, 
that  one  automatically  won  out.  While  this  method  seemed 
to  have  advantages,  some  inconsistancies  still  showed  up. 

The  way  we  finally  decided  to  compare  methods  was  to 
specify  each  method  as  a  certain  type  of  bound  as  discussed 
above,  and  then  compute  the  average  standard  deviation,  over 
the  set  of  simulations,  for  each  method,  where  the  standard 
deviations  compared  were  those  computed  for  the  type  of  bound 
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which  the  method  was  specified.  These  averages  could  be  com¬ 
pared  from  method  to  method  with  smaller  values  indicating 
a  better  method.  Thus,  a  method  which  produced  a  6=.01  bound 
could  be  better  than  another  method  which  produced  an  exact 
bound  simply  because  on  the  average  it  produced  results  which 
were  closer  to  the  true  system  reliabilities.  If  a  method 
did  not  produce  any  type  of  bound,  it  automatically  lost  out 
to  any  method  which  did.  It  could  however,  be  compared  to 
other  methods  which  did  not  produce  any  type  of  bound.  In 
the  next  sections,  we  report  the  results  we  accumulated. 

III.  SERIES  RESULTS 

In  our  previous  work  \J  >  we  found  that  the  results 
differed  between  the  case  of  series  systems  with  equal  compon¬ 
ent  test  sample  size  and  the  case  of  unequal  sample  sizes. 

The  LRS13  method  showed  great  promise  for  the  unequal  sample 
size  case  with  noticeably  smaller  sample  standard  deviations 
about  the  true  reliability  than  the  other  methods  available. 
However,  LRS13  was  so  optimistic  that  it  did  not  often  produce 
exact  90%  bounds  and  instead  was  most  consistently  a  .02- 
exact  90%  bound.  In  the  equal  sample  size  case,  this  was  even 
more  pronounced  with  LRS13  often  failing  to  be  any  type  of 
90%  bound.  As  mentioned  above,  LRS13  used  a  limited  trans¬ 
lation  version  of  Stein’s  estimator  to  adjust  the  test  data 
before  feeding  it  into  the  likelihood  ratio  method  and  this 
version  has  an  80%  level  of  savings.  In  [4],  tables  are 
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given  for  the  constant,  D,  necessary  to  achieve  six  levels 
of  savings:  .5,  -6,  .75,  .8,  .9,  .95  and  the  different  levels 
of  savings  offered  hope  for  improvements  in  LR313. 

Using  these  levels  of  savings,  six  methods,  LR2S1,  LRS32, 
LRSS3,  LRSS4 ,  LRSS5,  and  LRSS6,  were  developed  and  tested 
for  a  variety  of  cases.  The  set  of  simulations  covers  the 
cases  with  k=6,  10  or  20  components;  system  reliability  of 
.85,  .88,  .90,  .93,  and  (for  k=6)  .95;  and  equal  and  unequal 
sample  sizes.  For  comparison  purposes,  we  looked  first  at  the 
percentage  of  time  the  method  was  cumlatively  a  6*. 00,  .01  and 
.02  bound,  and  then  looked  at  the  corresponding  average  sample 
5-standard  deviation  for  each  method.  These  results  are 
grouped  by  system  size  and  type  of  sample  size  in  Tables  1-6. 
Tables  1-3  contain  the  results  for  unequal  sample  sizes 
(5,10,20,30),  (15,30,60),  and  (20,30,40,50).  Tables  4-6 
contain  the  results  for  equal  sample  sizes  (15,20)  and  (30,40). 


r - 

LRSSl 

LRSS4 

LRSS5 

LRSS6 

Exact 

6 

(  .1425) 

8 

( .1417) 

8 

( .1407) 

.01 

9 

(.1505) 

9 

( .1501) 

9 

( .1498; 

10 

( .1502) 

10 

(.1492) 

.02 

© 

(.1589)! 

© 

(.1585) 

km 

( .*§84) 

(.@89) 

© 

( .1579) 

Not 

)  a  Bound 

1 

1 

1 

1 

0 

0 

1 

!  Rank 

6 

b 

L _ _ _ _  .  - 

2 

2 

_ 

5 

1 _ 

1 

TABLE  1:  k=6 

Unequal  Sample  Sizes 


i 
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LRSS1 

Exact 

5 

( .1461) 

.01 

8 

(.1542) 

.02 

( .1625) 

Not 

a  Bound 

2 

Rank 

6 

TABLE  2:  k=10 

Unequal  Sample  Sizes 


LRSS1 


5  5  I  7  ® 

( .  1434)  ( .  1445)|(  .1488)  ( .  1518) 


(  .1515) (.1527) 


LRSS5  LRSS6 


(  5?6  4  0 ) 


9  9 

(.  1705) (• 1732 ) 


1 

9 

( .1695) 

9 

(.1798) 

0 

0 

2 

1 _ 1 

TABLE  3:  k=20 

Unequal  Sample  Sizes 

9-  Designated  Type  of  Bound  (90/5) 

-  Not  Really  a  .02  Bound 
(  )  -  Average  Standard  Deviation 


In  Table  1,  the  results  for  the  unequal  sample  size 
case  and  k=6,  we  can  see  that  LRSS6  is  the  preferable  method 


t 
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since  it  is  a  .02-exact  bound  12  out  of  12  times,  the  other 
methods  are  also  .02-exact  bounds,  and  the  average  sample 
.02-standard  deviation  of  LRSS6  is  the  smallest;  however, 
there  is  little  difference  among  these  averages.  In  Table  2, 
the  unequal  sample  case  for  k=10,  LRSS6  is  again  better  than 
the  other  methods  as  it  is  a  .01-exact  90%  bound  11  out  of 
12  times  and  its  average  sample  .01-standard  deviation  is 
smaller  than  the  allowable  average  sample  standard  deviation 
for  any  of  the  other  methods.  In  this  case  there  is  a  sig¬ 
nificant  difference  between  the  6-standard  deviations  of  the 
best  and  worse  methods.  In  Table  3,  the  unequal  sample  case 
for  k=20,  LRSS3  and  LRSS4  are  the  top  ranked  methods  with 
LRSS5  and  LRSS6  appearing  overly  conservative  but  definitely 
exact  90%  bounds.  Thus,  we  can  recommend  the  use  of  LRSS6 
(95%  savings)  in  all  unequal  sample  size  cases  with  the  pos¬ 
sibility  of  improving  the  results  if  LRSS4  is  used  instead 
for  large  (k>20)  systems. 

For  the  case  of  equal  sample  sizes,  the  results  of  the 
previous  report  were  completely  different,  with  the  case  of 
equal  sample  size,  n^l5  or  20,  being  different  from  the  case 
with  larger  sample  sizes,  n=3Q  or  40  .  The  results  in  Tables 
4-6  are  for  equal  sample  sizes  and  are  in  two  parts  for  each 
k,  part  A  is  for  ni=15  and  20,  while  part  B  is  for  n^=30  and 
40;  the  system  reliabilities  are  the  same  as  for  the  Tables 
1-3. 
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Equal  Sample  Sizes 


LRSS1 

LRSS2 

LRSS3 

LRSS4 

LRSS5 

LRSS6 

Exact 

© 

(  .1227) 

© 

(.1178) 

(.1?96) 

© 

(.1062) 

5 

( .0982) 

5 

(.0930) 

.01 

8 

( .1313) 

7 

(.1264) 

7 

(.1180) 

7 

( .1145) 

© 

(.1061) 

5 

(.1005) 

.02 

8 

( .1401) 

ini! 

8 

(.1266) 

7 

(.1230) 

7 

(.1144) 

© 

( . 1086) 

Not 

a  Bound 

0 

0 

0 

j__  .1 

i 

1.1  . 

1 

Rank 

4 

3 

1 

2 

1 

1 

5 

1 

l  6 

TABLE  4A:  k=6 


— 

LRSS1 

LRSS2 

LRSS3 

LRSS4 

LRSS5 

LRSS6 

Exact 

2 

( .0777) 

2 

(.0750) 

2 

(.0707) 

2 

( .0691) 

2 

( .0658) 

2 

( .0642) 

.01 

6 

( .0856) 

5 

(.0826) 

3 

( .0780) 

3 

(.0763) 

3 

( .0727) 

2 

(.0707) 

.02 

© 

(.0937) 

© 

(.0907) 

& 

(.0859) 

G* 

( .0841) 

<3> 

( .0801) 

(4> 

(.0781) 

Not 

a  Bound 

0 

0 

2 

3 

i ....  4. . 

4 

Rank 

2 

1 

3 

4 

6 

5 

TABLE  4B:  k=6 


O  -Designated  Type  of  Bound  (90%) 
*  -Not  Really  a  .02  Bound 
( )-Average  Standard  Deviation 


TP.. CTT. 'V  1UT.’ — ti  ■  *>.  jr 
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Equal  Sample  Sizes 


LRSS1 

LRSS2 

LRSS3 

LRSS4 

LRSS5 

LRS36 

Exact 

5 

(.1133) 

4 

(.1057) 

4 

C.Q926) 

4 

( .0871) 

3 

(.0730) 

3 

(.0647) 

.01 

<J) 

( . 1206) 

5 

(.1137) 

5 

(..  10Q3) 

4 

(.0945) 

4 

( .0782) 

3 

(.0701) 

.02 

7 

(.1291) 

( D 

( .1221) 

CD* 

( .  1083) 

& 

(.1023) 

q? 

( . 0866) 

0* 

(.0764) 

Not 

a  Bound 

1 

1 

2  ' 

■  -  1 

3 

4 

— 

Rank 

2 

1 

1 

3 

4 

6 

5  ___  \ 

TABLE  5A:  k=10 


LRSS1 

LRSS2 

'  LRSS3 

LRSS4 

LR3S5 

- 7 — 

LR3S6 

Exact 

3 

(.0769) 

2 

( .0722) 

(.0644) 

0 

( .0616) 

0 

(.0562) 

0 

( .0541) 

.01 

5 

( .0844) 

4 

(.0797) 

1 

(.0710) 

1 

(.0679) 

0 

( .0612) 

(.0580) 

.02 

0 

( . 0924) 

O 

( .0880) 

0* 

( .0984) 

O 

(.0749) 

CP* 

(.0674) 

& 

t . 0634) 

Not 

a  Bound 

1 

2 

3 

5 

7 

T  .  _  ] 

Rank 

1 

2 

3 

_ a _ 1 

6 

t 

> 

i 

5  1 

TABLE  5B :  k=10 


9  -Designated  Type  of  Bound  (90$) 
-Not  Really  a  .02  Bound 
()-Average  Standard  Deviation 
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Equal  Sample  Sizes 


Lrssi 

LRSS2 

LRSS3 

LRSS4 

LRSS5  !lRSS6 

Exact 

3 

(.10  30) 

1 

( . 0454 ) 

.01 

o 

VO  L>J 

O 

3 

(.0823) 

2 

(.0607) 

1 

(.0478) 

.02 

5 

( .1294) 

5 

(.1190) 

m 

J &> 

(p 

( .  0666) 

(D» 

(.0516) 

1 

l 

i 

3 

3 

4 

Rank  ! 

2 

1 

3 

5 

* 

6 

TABLE  6A:  k=20 


LRSSI 

LRSS2 

LRS3  3 

LRSS4 

Exact 

ISH 

0 

(.0554) 

0 

( .0504) 

0 

( .0414) 

0 

( .0414) 

.01 

3 

(.0796) 

2 

(.0745) 

1 

( .0611) 

i 

(.0552) 

0 

( .0428) 

0 

(.0397) 

.02 

( .0^87) 

( .0821) 

G> 

( .0678) 

A 

(J> 

(.0463) 

(.  040  3) 

Not 

a  Bound 

2 

2 

1 

3 

5 

5 

6 

Rank 

2 

—  1 - i 

3 

i  5 

:  « 

! 

6 

TABLE  6B :  k=20 


O  “Designated  Type  of  Bound  (90%) 
*  -Not  Really  a  .02  Bound 
()-Average  Standard  Deviation 
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Again,  the  results  for  these  cases  are  different  than 
that  for  the  unequal  sample  size  case  since  we  can  see  in 
Tables  4-6  that  the  higher  the  level  of  savings,  the  smaller 
the  average  sample-standard  deviation  when  everything  else 
is  constant.  In  most  cases  there  is  a  large  difference: 
for  k=10,  n=15  and  20,  6=.02,  the  average  sample  standard 
deviations  are  .1291,  .1221,  .1083,  .1023  .0866,  and  .0764. 
Unfortunately,  increasing  the  level  of  savings  has  an  adverse 
effect  on  the  ability  of  the  method  to  achieve  a  6-exact  bound: 
the  higher  the  level  of  savings  the  more  times  the  method 
fails  to  be  each  of  the  types  of  bounds.  In  many  cases  only 
LRSS1  and  LRSS2  achieve  any  type  of  bound  over  the  set  of 
simulations.  For  large  sample  size  cases,  this  pattern  was 
especially  pronounced  as  LRSS1  was  a  . 02-exact  bound  19  out 
of  22  times. 

Thus,  for  systems  which  will  be  tested  with  equal  sample 
sizes  for  all  components  we  recommend  that  LRSS1  or  LRSS2 
be  used  in  preferrence  to  the  other  methods. 

IV.  PSEUDO-SERIES  RESULTS 

For  the  pseudo-series  cases,  we  have  compared  five  methods 
of  estimating  lower  confidence  bounds,  four  likelihood  ratio 
methods  and  the  modified  log-gamma  method  (MLGB) .  Three  of 
the  likelihood  ratio  methods,  BIN9,  STN13  and  LRLOCK,  as  well 
as  MLGB,  were  used  for  the  series  case  (see  [7])  and  were 
discussed  earlier  in  this  report  as  well  as  in  [7] .  Recall 
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that  the  method  LRLOCK  adjusts  the  number  of  failures  of  any 

component,  1,  which  has  failed  zero  tests  by  letting  the 

number  of  failures  be  the  fraction  1/^kn^,  where  k  is  the 

number  of  components  in  the  system,  and  n,^  is  the  number  of 

times  the  component  was  tested.  The  fourth  likelihood  ratio 

method  we  considered  was  LRL2,  which  replaces  k  in  the  above 
k 

formula  with  I  a.  where  a.  is  the  exponent  of  the  i-th  com- 
1=1  i  i 

ponent  in  the  pseudo-series  system.  (For  the  series  case, 
all  ai»l,  so  this  method  would  be  identical  to  LRLOCK.) 

The  results  from  simulations  for  the  pseudo-series  reli¬ 
ability  structures  are  presented  in  the  followin  tables. 

Here  we  have  considered  two  cases:  the  case  with  all  compon¬ 
ents  in  a  system  being  from  one  particular  set  of  manufacturers 
(one  subpopulation  case);  the  case  with  some  of  the  components 
in  the  structure  having  been  redesigned  or  having  been  man¬ 
ufactured  by  two  distinct  processes  or  manufacturers  (two 
subpopulation  case) . 

For  both  the  one  subpopulation  and  the  two  subpopulation 
cases,  we  have  considered  the  situation  where  system  reliabil¬ 
ity  is  of  the  form 


R=pip2p3p4p5p6p7a, 

where  a  has  a  value  of  2.5,  4.5,  or  7-  The  corresponding 
test  sample  sizes  i=l ,2 , 3 , . . . , 7  are 


10 

(2a+l) xlO 


for  1*1,2, 3, ... ,6 
for  1*7 


When  there  are  two  subpopulations. 


components  which  have  not 
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been  redesigned  have  sample  sizes  of  2n.  ,  where  n<  is  as 
defined  above.  Those  components  which  have  been  redesigned 
each  have  sample  sizes  of  n^.  For  both  the  one  and  two  sub¬ 
population  cases,  the  results  have  been  grouped  into  system 
reliability  levels  of  .75-. 8,0,  .80-.85,  .85-. 90,  and  .90-. 95. 
Since  we  observed  no  significant  differences  for  the  differ¬ 
ent  a's,  we  did  not  separate  the  results  as  to  different  values 
of  a.  In  the  two  subpopulation  case,  the  proportion  of  the 
total  population  with  each  version  of  the  redesigned  component 
(or  components)  was  an  input  parameter.  Since  we  saw  no 
noticeable  differences  due  to  particular  proportions,  we  did 
not  separate  our  results  by  the  values  of  the  proportions 
in  each  set  of  simulations.  The  tables  indicate  the  number 
of  simulation  sets  in  which  each  of  the  five  methods  produced 
exact,  6®. 01,  <$  =  .02  bounds,  and  how  many  times  each  method 
was  not  any  of  these.  The  small  numbers  near  the  circled  ones 
indicate  the  average  values  of  the  standard  deviations  over 
the  simulation  sets. 

Case  I:  1  Subpopulation 
a)  Reliabilities  .75-. 80,  12  simulation  sets 


Rank 


2 


3 


5 


Pfege  20 


Case  I:  1  Subpopulation 
b)  Reliabilities  .80-. 85,  13  simulation  sets 


LRLOCK 

LRL2 

STN13 

BIN  9 

— 

MLGB 

Exact 

©1 

(  .216) 

8 

1 

1 

1 

.01 

13 

10 

1 

1 

1 

.02 

13 

0 

(  .227) 

0 

(  .202) 

(  .2^6) 

@ 

(.208) 

Not 

a  Bound 

12 

11 

12 

Rank 

1 

2 

4 

3 

5 

c)  Reliabilities  .85-. 90,  13  simulation  sets 


LRLOCK 

LRL2 

STN13 

BIN9 

MLGB 

Exact 

0 

(  .211) 

0 

(.203) 

9 

10 

1 

.01 

13 

13 

(.§4) 

( .^7) 

7 

.  .02 

13  ! 

13 

13 

13 

■E^l 

Not 

a  Bound 

4 

Rank 

5 

3 

1 

2 

1 _ i 

4 

Q  -Designated  Type  of  Bound  (90%) 
•  -Not  Really  a  .02  Bound 
( )-Average  Standard  Deviation 


mm  ft.*.  L  Vufcu  »  >  Vv>  ’fc  h  ^  . 


■  ?«■#*** 


Case  I:  1  Subpopulation 


If  a  person  wants  to  predict  a  90 %  lower  confidence  bound. 


and  he  has  a  system  similar  to  the  one  from  which  these  re¬ 
sults  were  gathered,  he  should  use  the  results  presented  to 
determine  which  of  these  methods  seems  most  useful.  Which 
table  to  use  is  determined  by  the  insight  the  user  has  on  the 
reliability  of  the  system.  If  he  feels  for  instance,  that 
the  system's  true  reliability  is  between  .85  and  .90,  he  could 
use  Table  b)  to  see  that  the  method  STN13  should  be  used  on 
his  data  as  a  <5=.01  bound.  If  however,  the  engineer  could 
only  say  that  he  felt  the  true  reliability  was  between  .75 
and  -95,  he  would  have  to  use  Table  g) ,  which  indicates  that 
LRLOCK  should  be  used  as  an  exact  bound. 

Some  information  is  lost  in  the  second  example  above. 
Notice  in  Tables  e)  and  f ) ,  that  LRLOCK  appears  to  be  the 
best  method  to  use  for  the  reliabilities  from  .75  to  .35,  but 
for  reliabilities  between  .85  and  .95,  LRL2 ,  STN13  and  BIN9 
appear  to  be  better  methods.  Thus  we  see  what  information 
may  be  lost  when  a  rougher  guess  of  the  system's  reliability 
must  be  made. 

The  following  tables  will  give  the  results  for  the  two 
subpopulation  case. 
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Case  I:  1  Subpopulation 

f)  Combined  Reliabilities  .85-. 95,  25  simulation  sets 


LRLOCK 

LRL2 

STN13 

BIN  9 

MLGB 

Exact 

0 

( .203) 

0 

(  .203) 

21 

22 

1  3 

.01 

25 

25 

0 

(  .19*0 

0 

(  .186) 

19 

.  02 

25 

25 

25 

25 

0* 

( .188) 

Not 

a  Bound 

4  ! 

i  Rank 

«  ; 

1 _ 1 _ _ 

i  2  . 

1  ■  ir-  ■ 

1 

i 

- 5-4 

g)  Combined  Reliabilities  .75-- 95,  50  simulation  sets 


STN13 

BIN  9 

MLGB 

Exact 

0 

(.211) 

mm 

28 

30 

16 

.01 

50 

0 

(.212) 

33 

32 

24 

.  02 

50 

49 

©* 

(.193) 

0* 

(.197) 

(.200) 

Not 

a  Bound 

15 

15 

i 

19 

Rank 

1 

2 

3 

4 

1  5 

9  -Designated 
-Not  Really 


Type  of  Bound  (90%) 
a  .02  Bound 


()-Average  Standard  Deviation 
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Case  II:  2  Subpopulations 
a)  Reliabilities  .75-. 80  ,  12  simulation  sets 


LRLOCK 

LRL2 

3TN13 

BIN9 

■ 

MLGB 

1  -  - 

Exact 

10 

9 

4 

5 

7 

.01 

© 

( .166) 

o 

(.3$4) 

7 

10 

10 

.02 

12 

12 

O 

(.tS9) 

(.§5) 

0 

( . 170) 

Not 

a  Bound 

1 

1 

1 

1 

1 

1 

Rank 

"  —  i 

4 

2 

_ i  ! 

!  3  ' 

5 

b)  Reliabilities  .80-. 85,  12  simulation  sets 


LRLOCK 

— 

LRL2 

STN13 

BIN  9 

MLGB 

Exact 

8 

7 

3 

5 

6 

.01 

e 

( .166) 

10 

6 

7 

9 

.  02 

12 

0 

(.164) 

0* 

(.159) 

HU 

<f<)# 

(.170) 

Not 

a  Bound 

2 

2 

2 

Rank 

2 

1 

3 

4 

1  5 

Q  -Designated  Type  of  Bound  (90%) 
*  -Not  Really  a  .02  Bound 
()-Average  Standard  Deviation 


,, ..  nyptlft  *f 1 
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Case  II:  2  Subpopulat ions 
c)  Reliabilities  .85-. 90,  17  simulation  sets 


LRLOCK 

LRL2 

STN1 3 

MLGB 

Exact 

10 

9 

8 

oo 

1 

.01 

11 

12 

8 

10 

6 

.  02 

© 

( .154) 

© 

( .152) 

(.140) 

<&, 

© 

(.146) 

Not 

a  Bound 

Rank 

5 

4 

i _ 1..J 

1 

!  2 

...  3 

d)  Reliabilities  .90-. 95,  12  simulation  sets 
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Case  II:  2  Subpopulations 

e)  Combined  Reliabilities  .75-. 85,  24  simulation  sets 


LRLOCK 

LRL2 

stni3 

mmm 

MLGB 

Exact 

18 

16 

7 

10 

13 

1 — 1 
O 

• 

0 

(.163). 

21 

13 

_  _j 

17 

19 

.02 

24 

m 

mm 

mm 

( .157) 

Not 

a  Bound 

3 

3 

3 

Rank 

1 

2 

3 

- 

4 

1 

5  1 

f)  Combined  Reliabilities  .85-. 95,  29  simulation  sets 


LRLOCK 

LRL2 

STN13 

BIN  9 

MLGB 

Exact 

21 

16 

14 

15 

1 

.01 

22 

22 

15 

18 

8 

.  02 

0 

(.148) 

(.1^5) 

0 

(.133) 

r  0, 

(.136) 

m 

Not 

a  Bound 

1 

2 

3 

0 

C. 

6 

Rank 

4 

3 

1 - 

1 

1 

2 

5 

")  -Designated  Type  of  Bound  (90%) 
*  -Not  Really  a  .02  Bound 
() -Average  Standard  Deviation 


3f VJjy  OT\T*  = 
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Case  II:  2  Subpopulations 
g)  Reliabilities  .  7 5— -95,  53  simulation  sets 


LRLOCK 

LRL2 

3TN13 

BIN9 

MLGB 

Exact 

38 

32 

21 

25 

14 

.01 

46 

43 

28 

35 

27 

.02 

( .So, 

mm 

KH 

@ 

(.149) 

1 

Not 

a  Bound 

1 

2 

6 

5 

9 

Rank 

3 

_ 2 _ 

4 

1 

5 

.^-Designated  Type  of  Bound  (90%) 

*  -Not  Really  a  .02  Bound 
() -Average  Standard  Deviation 

These  tables  are  used  just  as  those  for  the  one  subpop¬ 
ulation  case. 

Let  us  now  look  at  what  problems  may  occur  if  a  user 
follows  the  tables  after  making  an  incorrect  guess  of  the 
system  reliability.  If  we  choose  a  method  which  works  (pro¬ 
duces  a  90 %  bound  a  high  percent  of  the  time)  both  at  the 
true  system  reliability  and  at  the  guessed  reliability,  we 
will  get  a  valid  bound.  If  the  method  works  at  the  true  re¬ 
liability  level  of  the  system,  but  we  would  have  used  another 
method  had  we  made  a  correct  guess,  we  will  be  worse  off 
because  another  method  may  have  predicted  a  bound  which  is 
closer  to  the  true  reliability.  An  error  could  be  made  however. 
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method  was  seen  not  to  produce  a  true  90%  lower  bound 
e  correct  system  reliability.  Thus,  it  may  be  useful 
ake  a  close  look  at  the  reasoning  behind  predicting  the 
tern's  reliability. 


